Compressive Spectral Clustering

Spectral clustering has become a popular technique due to its high performance in many contexts. It comprises three main steps: create a similarity graph between N objects to cluster, compute the first k eigenvectors of its Laplacian matrix to define a feature vector for each object, and run k-means on these features to separate objects into k classes. Each of these three steps becomes computationally intensive for large N and/or k. We propose to speed up the last two steps based on recent results in the emerging field of graph signal processing: graph filtering of random signals, and random sampling of bandlimited graph signals. We prove that our method, with a gain in computation time that can reach several orders of magnitude, is in fact an approximation of spectral clustering, for which we are able to control the error. We test the performance of our method on artificial and real-world network data.Comment: 12 pages, 2 figure

Coherence-based Partial Exact Recovery Condition for OMP/OLS

We address the exact recovery of the support of a k-sparse vector with Orthogonal Matching Pursuit (OMP) and Orthogonal Least Squares (OLS) in a noiseless setting. We consider the scenario where OMP/OLS have selected good atoms during the first l iterations (l<k) and derive a new sufficient and worst-case necessary condition for their success in k steps. Our result is based on the coherence \mu of the dictionary and relaxes Tropp's well-known condition \mu<1/(2k-1) to the case where OMP/OLS have a partial knowledge of the support

Dictionary Identification - Sparse Matrix-Factorisation via ℓ1\ell_1-Minimisation

This article treats the problem of learning a dictionary providing sparse representations for a given signal class, via $\ell_1$-minimisation. The problem can also be seen as factorising a \ddim \times \nsig matrix Y=(y_1 >... y_\nsig), y_n\in \R^\ddim of training signals into a \ddim \times \natoms dictionary matrix \dico and a \natoms \times \nsig coefficient matrix \X=(x_1... x_\nsig), x_n \in \R^\natoms, which is sparse. The exact question studied here is when a dictionary coefficient pair (\dico,\X) can be recovered as local minimum of a (nonconvex) $\ell_1$-criterion with input Y=\dico \X. First, for general dictionaries and coefficient matrices, algebraic conditions ensuring local identifiability are derived, which are then specialised to the case when the dictionary is a basis. Finally, assuming a random Bernoulli-Gaussian sparse model on the coefficient matrix, it is shown that sufficiently incoherent bases are locally identifiable with high probability. The perhaps surprising result is that the typically sufficient number of training samples \nsig grows up to a logarithmic factor only linearly with the signal dimension, i.e. \nsig \approx C \natoms \log \natoms, in contrast to previous approaches requiring combinatorially many samples.Comment: 32 pages (IEEE draft format), 8 figures, submitted to IEEE Trans. Inf. Theor

Constrained Overcomplete Analysis Operator Learning for Cosparse Signal Modelling

We consider the problem of learning a low-dimensional signal model from a collection of training samples. The mainstream approach would be to learn an overcomplete dictionary to provide good approximations of the training samples using sparse synthesis coefficients. This famous sparse model has a less well known counterpart, in analysis form, called the cosparse analysis model. In this new model, signals are characterised by their parsimony in a transformed domain using an overcomplete (linear) analysis operator. We propose to learn an analysis operator from a training corpus using a constrained optimisation framework based on L1 optimisation. The reason for introducing a constraint in the optimisation framework is to exclude trivial solutions. Although there is no final answer here for which constraint is the most relevant constraint, we investigate some conventional constraints in the model adaptation field and use the uniformly normalised tight frame (UNTF) for this purpose. We then derive a practical learning algorithm, based on projected subgradients and Douglas-Rachford splitting technique, and demonstrate its ability to robustly recover a ground truth analysis operator, when provided with a clean training set, of sufficient size. We also find an analysis operator for images, using some noisy cosparse signals, which is indeed a more realistic experiment. As the derived optimisation problem is not a convex program, we often find a local minimum using such variational methods. Some local optimality conditions are derived for two different settings, providing preliminary theoretical support for the well-posedness of the learning problem under appropriate conditions.Comment: 29 pages, 13 figures, accepted to be published in TS

Accelerated Spectral Clustering Using Graph Filtering Of Random Signals

We build upon recent advances in graph signal processing to propose a faster spectral clustering algorithm. Indeed, classical spectral clustering is based on the computation of the first k eigenvectors of the similarity matrix' Laplacian, whose computation cost, even for sparse matrices, becomes prohibitive for large datasets. We show that we can estimate the spectral clustering distance matrix without computing these eigenvectors: by graph filtering random signals. Also, we take advantage of the stochasticity of these random vectors to estimate the number of clusters k. We compare our method to classical spectral clustering on synthetic data, and show that it reaches equal performance while being faster by a factor at least two for large datasets

Recipes for stable linear embeddings from Hilbert spaces to R^m

We consider the problem of constructing a linear map from a Hilbert space $\mathcal{H}$ (possibly infinite dimensional) to $\mathbb{R}^m$ that satisfies a restricted isometry property (RIP) on an arbitrary signal model $\mathcal{S} \subset \mathcal{H}$. We present a generic framework that handles a large class of low-dimensional subsets but also unstructured and structured linear maps. We provide a simple recipe to prove that a random linear map satisfies a general RIP on $\mathcal{S}$ with high probability. We also describe a generic technique to construct linear maps that satisfy the RIP. Finally, we detail how to use our results in several examples, which allow us to recover and extend many known compressive sampling results

Linear embeddings of low-dimensional subsets of a Hilbert space to Rm

International audienceWe consider the problem of embedding a low-dimensional set, M, from an infinite-dimensional Hilbert space, H, to a finite-dimensional space. Defining appropriate random linear projections, we propose two constructions of linear maps that have the restricted isometry property (RIP) on the secant set of M with high probability. The first one is optimal in the sense that it only needs a number of projections essentially proportional to the intrinsic dimension of M to satisfy the RIP. The second one, which is based on a variable density sampling technique, is computationally more efficient, while potentially requiring more measurements

Harmonic Decomposition of Audio Signals with Matching

We introduce a dictionary of elementary waveforms, called harmonic atoms, that extends the Gabor dictionary and fits well the natural harmonic structures of audio signals. By modifying the &quot;standard&quot; matching pursuit, we define a new pursuit along with a fast algorithm, namely the Fast Harmonic Matching Pursuit, to approximate N-dimensional audio signals with a linear combination of M harmonic atoms. Our algorithm has a computational complexity of O(MKN), where K is the number of partials in a given harmonic atom. The decomposition method is demonstrated on musical recordings, and we describe a simple note detection algorithm that shows how one could use a harmonic matching pursuit to detect notes even in di#cult situations, e.g., very di#erent note durations, lots of reverberation, and overlapping notes

Reconciling “priors ” and “priors ” without prejudice?

Abstract—We discuss a long-lasting qui pro quo between regularizationbased and Bayesian-based approaches to inverse problems, and review some recent results that try to reconcile both viewpoints. This sheds light on some tradeoff between computational efficiency and estimation accuracy in sparse regularization. I